Biomedical Test Collection with Multiple Query Representation

The objective of this study is to validate pseudo gold standards using multiple queries for biomedical document collection. Aspect query is a quasi similar query of the original query text which has the same information needs. Basic idea is that single information need can be represented in various expressions. Four aspect queries per one original query were created manually by fifteen numbers of college students. By collecting the top ranked documents in retrieved sets generated by various queries, aspect query based pseudo gold standard is generated. In order to demonstrate its feasibility, we compared rank ordering based on human judgment and pseudo judgments. Experimental results showed the high correlations of 0.863 (p<0.01). Any difference among query worker group was observed and background knowledge about query topic did not seem to be prerequisite to create query sentences. Through the involvement of the experiment, we concluded that the method using multiple aspect queries can be a promising method for building relevance judgment without human experts.